The Role of Design Experiments and Invariant Measurement Scales in the Development of Domain Theories

OVERVIEW In order to explain the concepts in the title, this paper refers to four other papers in this volume. It builds on the introduction to domain theory and validity-centered design in Bunderson(2002), and adds to that paper information about the need for and nature of principled design experiments and theory-based calibrations. These tools make it possible to construct, over time and over repeated cycles, the validity argument for a domain theory and its associated construct-linked scales. This article refers to the paper reporting work-in-progress on the development of a domain theory in English as a second language (ESL) by Diane Strong-Krause (2002). It provides background for that paper by discussing the need for theory-based calibrations in the ESL speaking domain. This need is great, because of the multi-semester nature of the domain, and the ever-incomplete nature of the sample available to the investigator in the environment of the Brigham Young University (BYU) English Language Center. This paper also provides some context for the paper by Newby, Conner, Grant, and Bunderson (2002). Principled Design Experiments as described herein require several types of measurement scale invariance. IRT models in general offer invariance across different samples of people and sets of items. Beyond that, it has been argued that the Rasch model provides equal interval scales through its link to Additive Conjoint Measurement (ACM). The analysis in the Newby, et al. paper shows that the mathematical connection between the Rasch model and additive conjoint structures is stronger than had previously been proved. Armed with this stronger proof, a simulation study was defined in the hopes of showing that the Rasch model could return known location parameters from unidimensional, equal interval scales, cutting through any independent multidimensional error, and thus providing the best basis for constructing construct-linked scales. Such error is due to the existence of different degrees of item uniqueness and sample error. Together, these two sources of error yield different slopes in the " a " parameter. The simulation study is reported in Pelton (2002). As is often the case in research, hopes are not always borne out. In Pelton's calibrations of simulation data where the true location parameters were known, the 2PL calibration program returned the true originating parameters and true order more closely than did the Rasch calibration program, especially when the distribution of students was shifted to the high end (as if they had progressed and learned). The …